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Jk  n  improvised  explosive  device  is  detonated  near  an  Iraqi 
marketplace.  A  U.S.  Army  unit  patrolling  the  vicinity  is  the 
M  M  obvious  target.  A  young  Iraqi  boy  and  three  women  are  killed 
instantly  in  the  explosion.  Dozens  of  Iraqi  citizens  are  injured.  A 
shopkeeper's  wife  saw  somethirig  suspicious.  CPT  Clipp  quickly  goes 
to  the  market  to  meet  with  the  shopkeeper  and  his  wife.  He  greets 
them  in  Arabic ,  then  asks  the  woman  to  tell  him  what  she  saw. 


The  new  C3IT-D,  when  fielded  Armywide,  will  better  prepare  Soldiers  cognitively  and  psychologically  for 
cultural  and  language  interaction  when  they  deploy  into  a  foreign  theater  of  operations.  Here,  SGT  Jonathan 
Ellis,  Alpha  Battery,  2nd  Battalion,  32nd  Field  Artillery  Regiment,  4th  Brigade  Combat  Team,  1st  Infantry 
Division,  pulls  security  during  a  patrol  in  the  Mansour  district  of  Baghdad,  Iraq,  on  July  31,  2007.  (U.S.  Army 
photo  by  SGT  Tierney  Nowland.) 
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The  shopkeeper  is  outraged.  “ You  speak  to 
me,  not  to  her!”  he  shouts.  Clipp  quickly 
apologizes,  rethinks  his  approach  and 
then  carefully  works  with  the  shopkeeper 
to  get  the  information  he  needs  from  the 
man’s  wife.  A  short  time  later,  a  suspect 
is  detained.  After  many  questions  about 
the  man’s  activities  that  morning,  Clipp 
asks  the  man,  “Did you  have  a  push  cart 
at  the  market  this  morning ?”  The  man 
replies,  “I  did  not  have  a  pottery  cart.  ” 
Clipp  did  not  mention  a  pottery  cart. 

The  detainee  just  made  a  big  mistake 
and  Clipp  knows  that  he  has  his  man. 


What  you  just  witnessed  is  not  a  real 
event,  but  rather  a  training  scenario 
based  on  a  real  event.  Soldiers  need 
training  in  culturally  appropriate  behav¬ 
ior,  bilateral  engagement  and  in  the 
art/science  of  tactical  questioning. 

Enter  human-oriented  training.  Cur¬ 
rently,  live  Iraqi  role  players  provide  real¬ 
ism  for  home  station  and  combat  train¬ 
ing  center  human-oriented  training. 


This  vignette,  however,  is  not  part  of  a 
live  scenario.  It  comes  from  the  Cultural 
&  Cognitive  Combat  Immersive 
Trainer-Demonstration  (C3IT-D)  proto¬ 
type  tested  at  Fort  Benning,  GA.  Clipp 
is  a  real  Soldier  but  all  of  the  other 
participants  are  life-sized  computer¬ 


generated  virtual  humans  created  by  the 
University  of  Southern  California’s 
(USCs)  Institute  for  Creative  Technolo¬ 
gies  (ICT).  The  real¬ 
istic  environment, 
complex  characters 
and  technologies  inte¬ 
grated  to  create  this 
prototype  were  the  re¬ 
sult  of  collaboration 
by  the  ICT;  the  U.S. 

Army  Research,  De¬ 
velopment  and  Engi¬ 
neering  Command 
(RDECOM)  Simula¬ 
tion  and  Training 
Technology  Center 
(STTC);  and  U.S. 

Army  Infantry  Center. 


In  1997,  the  National 
Research  Council  Re¬ 
port  Modeling  and 


the  Executive  Agent  for  this  UARC 
and  manages  the  research  contract  for 
the  Army. 


ICT’s  mission  is  to  “build 
a  partnership  among  the 
entertainment  industry, 
the  Army  and  academia, 
with  the  goal  of  creating 
synthetic  experiences  so 
compelling  that 
participants  react  as  if 
they  are  real.  The  result  is 
engaging,  new  immersive 
technologies  for  learning, 
training  and  operational 


environments. 


Simulation  [M&S]:  Linking  Entertain¬ 
ment  and  Defense  identified  opportuni¬ 
ties  for  leveraging  DOD  and  entertain¬ 
ment  research.  In  1999,  the  ICT  was 
chartered  as  a  University  Affiliated  Re¬ 
search  Center  (UARC)  to  conduct  re¬ 
search  in  M&S  technologies  by  leverag¬ 
ing  the  entertainment  industry’s, 
DOD’s  and  academia’s  research  and  in¬ 
novation.  RDECOM  STTC  serves  as 


The  ICT  has  evolved 
since  it  opened  in 
1999.  Opportunities 
to  leverage  ICT  re¬ 
search  have  expanded 
beyond  immersive 
training  technologies 
into  operational  capa¬ 
bilities.  Research  in 
the  areas  of  artificial  in¬ 
telligence  (AI),  realistic 
graphics  and  immer¬ 
sive  audio  have  signifi¬ 
cant  potential  to  en¬ 
hance  multimodal  dis¬ 
play  systems,  decision 
support  systems  and 
user  interfaces  in  next 
generation  command 


and  control  systems.  They  also  enable 
virtual  reality  immersion  therapy  to  treat 
Post-Traumatic  Stress  Disorder.  With 
this  in  mind,  ICT’s  mission  is  to  “build 
a  partnership  among  the  entertainment 
industry,  the  Army  and  academia,  with 
the  goal  of  creating  synthetic  experiences 
so  compelling  that  participants  react  as  if 
they  are  real.  The  result  is  engaging, 
new  immersive  technologies  for  learning, 
training  and  operational  environments.” 


Research  in  AI,  graphics  and  sound 
alone  is  not  sufficient  to  create  com¬ 
pelling  and  memorable  experiences. 
Another  key  element  to  immersion  is 
provided  by  well-crafted  stories  and 
characters,  which  draw  the  viewer  into 
the  scenario,  setting  the  conditions  for 
memorable  experiences.  Through  its 
collaboration  with  Hollywood  and  the 
game  industry,  ICT  is  conducting  re¬ 
search  to  extend  state-of-the-art  story 
development,  immersive  games  for  train¬ 
ing,  rapid  scenario  generation  and  sys¬ 
tem  production.  Realistic  environments 
and  well-crafted  scenarios  are  necessary 
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I GT:  A  Nexus  for  In  no  vat  ion 

Can  we  ma-Ae  learning  useful  and  unforgettable? 


to  create  truly  useful  experiences  but 
alone  are  insufficient.  Whether  one  is 
focused  on  training  a  task  or  learning  a 
concept,  it  is  crucial  to  present  the 
right  set  of  developmental  experiences. 
This  is  where  the  learning  sciences 
make  a  contribution  to  this  nexus  of 
innovation  —  through  design,  guid¬ 
ance,  mentoring  and  assessment. 

ICT’s  multidisciplinary  approach  to 
research  in  immersive  learning  tech¬ 
nologies  is  depicted  in  the  figure 
above.  The  end  goal  is  effective  learn¬ 
ing  that  is  useful  and  unforgettable. 

The  ICT  research  portfolio  spans  these 
areas  through  an  interdisciplinary  re¬ 
search  program  that  includes  the  fol¬ 
lowing  major  thrusts. 

Virtual  Humans 

Most  military  simulations  focus  on 
high-fidelity  models  and  simulators 
of  vehicles,  aircraft  and  units.  But 
mission  success  also  requires  human 
interaction,  whether  it  is  within  a 
team,  engagement  with  locals  or  work¬ 
ing  with  nongovernmental  organiza¬ 
tions.  To  train  these  human-oriented 
tasks,  the  objective  is  to  create  interac¬ 
tive  simulations  of  people  —  virtual 
humans.  A  significant  portion  of 
ICT’s  research  in  AI  is  focused  on 


solving  the  challenges  to  developing 
virtual  humans  that: 

•  Fully  perceive  their  environment 
(including  real  humans). 

•  Interact  using  the  full  repertoire  of 
human  interaction: 

0  Robust  verbal  language. 

°  Nonverbal  communication 
(gestures,  facial  expressions,  etc.). 

•  Manipulate  and  interact  with 
the  environment. 

•  Model  emotions  and  their  effect  on 
behavior. 

•  Can  introspect  and  model  other’s  be¬ 
liefs,  desires  and  intentions. 

•  Learn. 

While  the  C3IT-D  prototype  de¬ 
scribed  above  is  a  glimpse  into  how  a 
fully  capable  virtual  human  can  be 
used  in  a  training  system,  individual 
technologies  are  already  being  transi¬ 
tioned  to  Army  programs.  For  in¬ 
stance,  RDECOM  STTC  and  ICT  are 
currently  working  with  Program  Man¬ 
ager  (PM)  Constructive  Simulation  to 
integrate  natural  language  and  emo¬ 
tion  models  into  the  Intelligence  and 
Electronic  Warfare  Tactical  Proficiency 
Trainer.  This  successful  technology 
transition  resulted  in  a  $34  million 
program  cost  avoidance. 


Realistic  Graphics 

A  typical  animated  film  or  game  requires 
expensive  and  time-consuming  artistic 
effort.  The  ICT  graphics  lab  is  perform¬ 
ing  research  that  will  enable  simulations 
that  are  photorealistic,  rapidly  reconfig- 
urable  and  available  in  real-time. 

A  challenge  to  providing  photorealistic 
animated  images  is  the  need  for  realistic 
lighting  within  a  simulation.  The  graph¬ 
ics  lab  has  created  techniques  to  rapidly 
capture  and  manipulate  image  data, 
which  includes  the  reflectance  properties 
of  an  object  or  person.  These  techniques 
enable  rendering,  in  a  computer-gener¬ 
ated  environment,  of  photo-real  images 
of  objects  or  people  that  react  appropri¬ 
ately  to  the  lighting  conditions  in  the 
simulation  (streetlights,  explosions,  etc.). 
These  photo-real  images  can  be  calcu¬ 
lated  automatically  and  produce  consis¬ 
tent  and  realistic  appearance.  The  results 
of  this  research  have  been  exploited  in 
movies  such  as  Spider  Man  II,  Superman 
Returns  and  King  Kong.  A  new  challenge 
being  addressed  by  ICT  is  to  create 
highly  realistic  real-time  animations  of 
virtual  humans.  This  capability  will  en¬ 
able  mission-rehearsal-level  training  of 
human-oriented  tasks. 

Social  and  Cultural 
Representation 

Simulations  need  to  account  for  the  op¬ 
erational  environment’s  social,  cultural 
and  political  complexities.  This  goes  be¬ 
yond  developing  the  hard-coded  meth¬ 
ods  that  occur  each  time  a  new  area  of 
operations  is  introduced.  The  ICT  is  re¬ 
searching  methods  to  represent  cultural 
knowledge  in  simulated  characters  as 
swappable  culture  modules.  Critical  to 
this  effort  are  the  answers  to  some  very 
basic  questions:  What  does  such  a  cul¬ 
ture  module  contain?  Flow  does  the  sys¬ 
tem  use  it?  Can  culture  modules  be  au¬ 
thored?  The  idea  of  a  swappable  culture 
module  that  modifies  the  behavior  of 
simulated  characters  cannot  be  fully 
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exploited  without  also  addressing  the 
need  for  representing  these  characteristics 
within  the  environ¬ 
ment.  Key  to  the  de¬ 
velopment  of  rapid 
scenario  generation 
tools  is  developing  a 
fundamental  under¬ 
standing  of  how  to 
represent  the  social, 
cultural  and  political 
context  of  the  terrain, 
and  do  it  in  a  modular 
and  easy-to-author 
manner.  Advances  in 
this  research  will  en¬ 
able  simulation-based 
training  in  geo-specific 
terrain  that  has  been 
augmented  with  social,  cultural  and 
political  information. 

Intelligent  Tutoring  Systems 

The  ICT  is  exploring  the  integration  of 
Intelligent  Tutoring  System  technology 
into  human-oriented  training  systems. 
This  research  is  set  within  the  Guided 
Experiential  Learning  (GEL)  model  of 
training  design.  Created  by  Dr.  Richard 
Clark,  Professor  of  Educational  Psychol¬ 
ogy  and  Technology  at  the  USC  Rossier 
School  of  Education,  GEL  has  been 
adopted  as  a  supplement  to  Soldier 


training  by  the  U.S.  Army  Training  and 
Doctrine  Command  (TRADOC).  It 
emphasizes  deliberate 
practice  and  the  criti¬ 
cal  role  of  feedback 
and  guidance.  The 
ultimate  goal  is  to 
build  automated  tu¬ 
tors  and  coaches  that 
maximize  the  learning 
effectiveness  of  com¬ 
puter  simulations  used 
for  training. 

This  team  seeks  to 
answer  fundamental 
questions  that  face  all 
serious  games  and 
training  simulations: 

•  The  timing  of  feedback  —  how 
often  should  a  coach  intervene? 

•  Content  —  is  goal-based  feedback 
superior  to  explanation-based? 

•  Reflection  —  what  is  the  appropriate 
structure  for  an  after-action  review? 

The  results  of  this  learning  science  re¬ 
search  are  critical  to  ensure  that  the  in¬ 
vestment  in  the  simulation’s  realism 
and  the  complex  scenarios  are  coupled 
with  the  guidance  and  feedback  neces¬ 
sary  to  enable  effective  learning. 


The  ICT  has  developed  active  partner¬ 
ships  with  government  laboratories, 
other  universities,  Army  schools,  PMs 
and  industry  to  align  its  research  with 
the  greater  community’s  needs.  A  cur¬ 
rent  example  of  a  collaborative  effort  is 
the  Learning  With  Adaptive  Simula¬ 
tion  and  Training  Army  Technology 
Objective  (ATO),  which  is  a  team 
effort  among  the  ICT,  RDECOM 
STTC,  Army  Research  Institute,  Army 
Research  Laboratory  and  TRADOC. 
An  early  spin-out  from  this  ATO  is  the 
Bi-Lateral  Engagement  and  Negotia¬ 
tion  Training  System.  This  PC-based 
prototype  incorporates  many  of  the 
research  technologies  discussed  into  a 
GEL  system.  The  prototype  is  being 
tested  by  units  from  the  101st  Air¬ 
borne  Division,  1 0th  Mountain  Divi¬ 
sion,  1st  Infantry  Division  and  the 
School  for  Command  Preparation. 

ICT’s  early  vision  —  to  bring  together 
academia,  the  entertainment  industry 
and  the  Army  to  create  engaging,  new 
immersive  technologies  for  human- 
oriented  learning  and  training  —  has 
proven  to  be  even  more  important 
today  than  it  was  in  1999.  ICT  Exec¬ 
utive  Director  Dr.  Randy  Hill  wel¬ 
comes  the  opportunity  to  partner  with 
organizations  to  transition  ICT  re¬ 
search,  or  collaborate  to  create  innova¬ 
tive  multidisciplinary  solutions  to 
tough  immersive  learning  technology 
challenges. 


DR.  JEFF  G.  WILKINSON  is  an 

RDECOM  STTC  member.  He  is  the 
government’s  PM  for  the  ICT  at  USC. 

Prior  to  retiring  from  the  Army,  he  was  the 
Chief,  Simulations  Division,  Armor  Center, 
and  served  as  the  Deputy  TRADOC  System 
Manager  for  the  Combined  Arms  Tactical 
Trainer.  He  holds  an  M.S.  and  Ph.D.  in 
economics/operations  research  from  The 
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Trainers  at  the  101st  Airborne  Division  learn  how 
to  use  the  Bi-Lateral  Engagement  and  Negotiation 
Trainer  prototype.  (U.S.  Army  photo  by  MAJ 
Chris  Quick,  101st  Airborne  Division.) 


Whether  one  is  focused 
on  training  a  task  or 
learning  a  concept,  it  is 
crucial  to  present  the  right 
set  of  developmental 
experiences.  This  is  where 
the  learning  sciences  make 
a  contribution  to  this 
nexus  of  innovation  — 
through  design,  guidance, 
mentoring  and  assessment. 
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